The present invention relates to an image processing apparatus. More particularly, the invention relates to an image processing apparatus for calculating the amount of compensation for the effects caused by vibrations, a method for causing the apparatus to make the calculations, and a program for causing a computer to execute the method.
Portable video cameras have come into general use today. These cameras typically include a charge transfer type solid-state image pickup device exemplified by the CCD (Charge Coupled Device) sensor, or an X-Y address type solid-state image pickup device represented by the CMOS (Complementary Metal Oxide Semiconductor) sensor.
Driven on a single low voltage, the CMOS sensor consumes less power than the CCD sensor and is more suitable for integration with its peripheral circuits. These advantages have prompted researchers to consider adopting the CMOS sensor for use in the image processing apparatus such as video cameras.
However, it has been found difficult to use the CMOS sensor as the image pickup device of the video camera or other image processing apparatuses with a view to recording moving or still pictures of high quality. A major reason for that difficulty is the presence of distortions in pictures taken by the camera being held by shaky hands. In the case of the CCD sensor that has been prevalent as the image pickup device for the image processing apparatus, a single value calculated on the basis of shaky hand information obtained per field or per frame is used as the amount of compensation for minimizing the adverse effects on pictures caused by shaky hands. The use of a single value for shaky hand compensation is made possible by the fact that the period of exposure is equal for all pixels of the CCD sensor with no distortion in pictures resulting therefrom.
As opposed to the CCD sensor, the CMOS sensor is prone to picture distortions stemming from shaky hands because the CMOS device takes pictures of an object in a manner different from the way the CCD sensor operates, as outlined below.
The charge transfer type solid-state image pickup device such as the CCD sensor is capable of exposing all pixels at the same time in reading pixel data. By contrast, the X-Y address type solid-state image pickup device such as the CMOS sensor reads pixel data one pixel or one line at a time. While pixel data is being read, the time it takes for the pixel-reading solid-state image pickup device to move from one pixel to another per line is only negligible when compared with the time it takes for its line-reading counterpart to move from one line to another. Thus the argument that follows will only take the differences in time period between lines into consideration, as shown in FIGS. 1A and 1B.
Illustratively, FIG. 1A shows how a CMOS sensor reads all pixels at intervals of 1/T seconds, with each picture formed by lines #1 through #N. FIG. 1B depicts an example in which the period of exposure starts at time t1 and ends at time t2 for line #1, ends at time t3 for line #2, and ends at time t4 for line #3.
The time difference between t2 and t3 is given as Δt, and the difference between t3 and t4 also as Δt. That is, the exposure period involves the time difference of Δt per line. In this case, there arises a time difference close to 1/T seconds during exposure between line #1 at the top of the picture and line #N at the bottom. With such a difference in terms of exposure period taking place between line #1 and #N per picture, the exposure time difference per line contributes to letting the CMOS sensor held by shaky hands distort pictures of an object being imaged.
How pictures taken of the object are distorted is explained below by referring to FIG. 2. FIG. 2 shows examples of a rectangle 20 being taken as the object in picture 10. The picture in the middle shows a distortion-free object taken by an image processing apparatus not held by shaky hands. The normally imaged object is shown as the correctly formed rectangle 20.
The right-hand side picture in FIG. 2 shows a distorted object taken by an image processing apparatus shaken rightward. Similarly, the left-hand side picture indicates a distorted object taken by an image processing apparatus shaken leftward. The top picture in FIG. 2 illustrates a vertically elongated object taken by an image processing apparatus shaken upward; and the bottom picture presents a vertically compressed object taken by an image processing apparatus shaken downward.
As described, the CMOS sensor is subject to exposure period differences between lines (i.e., differences in imaging timing between lines). For that reason, it is impossible to fully compensate for the adverse effects of shaky hands by use of a single value calculated on the basis of shaky hand information obtained per field or per frame. The result is a cluster of distorted pictures being presented to the user.
Some image processing apparatus that compensate for the picture distortion caused by shaky hands have been proposed. One such apparatus is disclosed illustratively in Japanese Patent Application No. 2001-358999. This image processing apparatus is designed to change its read position per line to compensate for horizontal vibrations of the hands while changing vertically the position of each line from which to read in order to compensate for vertical vibrations of the hands.
The proposed technique above, however, fails to consider the limits of shaky hand compensation, to be discussed below. That failure can lead to unnaturally formed pictures. Some problems associated with the limits of shaky hand compensation will now be described with reference to FIGS. 3A and 3B. These two figures show what can take place because of the limits of compensation existing in the horizontal and vertical directions. With measures taken to compensate for the shaky hands per line, each figure illustrates relative positional relations between a picture data area held in a memory before compensation on the one hand, and a picture data area in effect after the compensation on the other hand.
FIG. 3A shows a state in which an original object 21 is imaged into a picture data area 11 provided in the memory but the obtained image is distorted into a picture 31 due to shaky hands. In this state, the amount of horizontal vibrations from the shaky hands is so large that a corner 41 of the picture 31 has overflowed the picture data area 11. That means the picture data subsequent to line #j in the picture data area 11 is not sufficient for subsequent compensation.
Likewise, FIG. 3B indicates a state where an original object 22 is imaged into a picture data area 12 provided in the memory but the obtained image is distorted into a picture 32 because of shaky hands. In this state, the amount of vertical vibrations from the shaky hands is so large that a portion 42 of the picture 32 has overflowed the picture data area 12. That means the picture data subsequent to line #k in the picture data area 12 is insufficient for subsequent compensation.
As described above, the CMOS sensor attempting to make shaky hand compensation without regard to the limits of the compensation could fail to provide full picture data following the compensation. The ultimately formed pictures tend to be distorted, giving viewers an awkward, uncomfortable impression.
The present invention has been made in view of the above circumstances and provides an image processing apparatus which, when compensating for the adverse effects on pictures caused by shaky hands or some other vibrating sources, does not overcompensate for the pictures exceeding the limits of the compensation.